Universal Context Propagation for Distributed System Instrumentation

Many tools for analyzing distributed systems propagate contexts along the execution paths of requests, tasks, and jobs, in order to correlate events across process, component and machine boundaries. _ere is awide range of existing and proposed uses for these tools, which we call cross-cutting tools, such as tracing, debugging, taint propagation, provenance, auditing, and resourcemanagement, but few of them get deployed pervasively in large systems.When they do, they are brittle, hard to evolve, and cannot coexist with each other. While they use very diòerent context metadata, the way they propagate the information alongside execution is the same. Nevertheless, in existing tools, these aspects are deeply intertwined, causing most of these problems. In this paper, we propose a layered architecture for crosscutting tools that separates concerns of system developers and tool developers, enabling independent instrumentation of systems, and the deployment and evolution ofmultiple such tools. At the heart of this layering is a general underlying format, baggage contexts, that enables the complete decoupling of system instrumentation for context propagation from tool logic. Baggage contexts make propagation opaque and general, while still maintaining correctness of themetadata under arbitrary concurrency and diòerent data types. We demonstrate the practicality of the architecture with implementations in Java and Go, porting of several existing cross-cutting tools, and instrumenting existing distributed systems with all of them.